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Abstract. We evaluate typical performance of irregular low-density generator-matrix 
(LDGM) codes, which is defined by sparse matrices with arbitrary irregular bit degree 
distribution and arbitrary check degree distribution, for lossy compression. We apply 
the replica method under one-step replica symmetry breaking (1RSB) ansatz to this 
problem. 
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1. Introduction 

The channel coding can be considered as the dual problem of lossy source coding in 
rate-distortion theory [JJ [2] . Matsunaga and Yamamoto showed that it is possible to 
approach the binary rate-distortion bound using LDPC codes [3]. In recent years, lossy 
source coding problem based on low-density generator-matrix (LDGM) codes is widely 
investigated. 

This scheme can attain high performance very close to the Shannon bound, however 
it needs solinving a combinatorial optimization problem to obtain optimal source 
coding. Some practical encoding algorithms are proposed for this scheme, e.g., a belief- 
propagation-based encoder proposed by Murayama [I] and a survey-propagation-based 
encoder proposed by Wainwright and Maneva [5]. 

Performance of this scheme is also explored by various approaches. Murayama 
and Okada applied replica methods to evaluate performance of LDGM codes defined 
by regular sparse matrices for lossy compression [6J. Ciliberti et al. have used the 
cavity method to evaluate check-regular LDGM performance [7J[8]. On the other hand, 
Martinian and Wainwright derived rigorous upper bounds on the effective rate-distortion 
function of LDGM codes for the binary symmetric source [9]. Dimakis et al. derived 
lower bounds for check-regular LDGM codes [TUl ITT] . 

With respect to irregular LDGM codes analyzed so far, elements of a reproduced 
message are given by exactly K elements chosen at random from a codeword. This 
implies that previous analyses treat only the case where a bit degree distribution is 
Poissonian. An irregular bit and check degree distributions of a generator matrix are 
not optimized for lossy source coding. The goal of this paper is to evaluate how typical 
performance of irregular LDGM codes for lossy compression depends on a bit degree 
distribution and a check degree distribution. 

2. Background 

Let us first provide the concepts of the rate-distortion theory [TJ. Let x be a binary 
i.i.d. source discrete which takes in a source alphabet X = {0, 1} with P[x = 0] = ¥[x = 
1] = 1/2, where P represents the probability of its argument. An source message of M 
random variables, x = t (xx, ■ ■ ■ ,xm) £ X M , is compressed into a shorter expression, 
where the operator * denotes the transpose. The encoder describes the source sequence 
x G X M by a codeword z = T{x) G X N . The decoder represents a; by a reproduced 
message x = Q(z) G X M . Note that M represents the length of a source sequence, 
while iV(< M) represents the length of a codeword. The code rate is R = N/M. The 
distortion between single letters is measured by the Hamming distortion defined by 



and the distortion between M-bit sequences x G X and x G X M is measured by 
the averaged single-letter distortion as d(x,x) = $^u=i ^( x /i> This results in 




(1) 



Typical performance of irregular LDGM codes for lossy compression 



3 



the probability of error distortion, since E[d(x, x)] = P[x 7^ x], where E represents the 
expectation. The distortion associated with the code is defined as D = K[d(x, x)], where 
the expectation is over the probability distribution on X x X . A rate distortion pair 
(R, D) is said to be achievable if there exists a sequence of rate distortion codes (J 7 , Q) 
with E[d(£E, a:)] < D in the limit M — > 00. The rate distortion function R(D)is the 
infimum of rates R such that (R, D) is in the rate distortion region of the source for 
a given distortion D. The rate-distortion function of a Bernoulli (1/2) i.i.d. source is 
given by 

R(D) = l-h 2 (D), (2) 
where h,2{x) = — xlog 2 (x) — (1 — x) log 2 (l — x) is the binary entropy function. 



3. Lossy Compression Scheme 

An source message of M random variables, x G X M , is compressed into a shorter 
expression, where the operator * denotes the transpose. The encoder describes the 
source sequence x G X M by a codeword z = J-'(x) G X N . The decoder represents x by 
a reproduced message x = Q(z) G X M . The code rate is R = N/M < 1. 

Using a given M x N sparse matrix A = (a^) G {0, 1 j MxAr ; the decoder is defined 

as 

Q{z) = Az (mod 2). (3) 
The encoding is represented by 

F(x) = aigmin d(x,Q(z)), (4) 

where d is the distortion measure. In this paper, we use the Hamming distortion. 
Although the definition means that a computational cost of the encoding is of 0(e N ), 
we can utilize some suboptimal algorithms based on message passing to encode (4],l5j. 



4. Analysis 

To simplify the calculations, we first introduce a simple isomorphism between the 
additive Boolean group ({0,1},©) and the multiplicative Ising group ({+1, — 1}, x) 
defined by J x J = (-1)*®* where J, J G {+1,-1} = J and x, x G {0,1} = X. 
Hereafter, we use the following Ising (bipolar) representations : the Ising source message 
J G J M , the Ising reproduced message J G J M and the Ising codeword £ G J N . The 
source bit can be described as a random variable with the probability: 

Pj(J) = \s(J-l) + ^5(J+l), (5) 

where 5(x) denotes Dirac's delta function. The /z-th element of the Ising reproduced 
message is given by products of the elements of the tentative Ising codeword s G J N : 
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where £(//) = {i\a^ = 1,A = (<v)}. 

The matrix A has nonzero elements in the /i-th row and Cj nonzero elements in 
the i-th column. We consider the source length and the codeword length to be infinite, 
while code rate R is kept finite. The parameter K x ■ ■ • K M and Ci, • • • , Cjv are usually 
of 0(N°), therefore the matrix A becomes very sparse. In densely constructed cases, 
we also assume that these parameters are not of 0(N°) but K,Ci, - ■ ■ , <C N holds. 
Counting the number of nonzero elements in the matrices leads to K% + • - • + Km = 
G\ + • ■ ■ + Cn- The code rate is therefore R = K/C, where K = Y^=x K-n ano ^ 
C = jr Yui=i Ci- Code constructions are described by the connectivity parameter 
... ; G {0, 1} which specifies a set of indices zi, • • • , corresponding to nonzero 
elements in the /i-th row of the sparse matrix A. The connectivity parameter is defined 
by 

h '~> iK » \ 0, otherwise ' 1 ' 

An ensemble of codes is generated as follows, (i) Sets of {K±, • • • Km} and 
{Ci, •••Cjv} are sampled independently from an identical distributions Pjc(K) and 
Pc(C), respectively, (ii) The connectivity parameters 2^...^ are generated such that 

M 

E E ^«.,-« M = ^ (8) 

where $^/ il=i ia ... ijf \ denote the summation over {(i 2 , ■ ■ • , 2r: m )g {1, ■ - • , A^}^ -1 |i 2 < 

• • • < iKpfa^i, ■ ■ -,iK^i}- 

To analyze typical performance of rate-compatible LDGM codes for lossy 
compression, we apply a analytical method similar to references [6j [121 131 HH US]- 
The Hamming distortion c?(«7, J) becomes d(J, J) — ~ — ^ S^Li ^ilLe/;^) s i}j since 
J, «/ G J7" M . Using the connectivity parameter ... ■ , we can rewrite this Hamming 
distortion in the form: 

1 1 - 

d(J ' J) = 2~2M^ E^ J A-«v (9) 

where ... ijf > denote the summation over {(z 1; • • • , ix^)^ {1 ; • ' * j < ■ ■ • < 

*r: m }- We here define the Hamiltonian 

H(s,J)=Md(J,J(s)), (10) 

to explore typical performance. The free energy is calculated from the partition function 
— J2sej N ex P[ — l37~L(s, J)}. From the free energy, we can obtain a distortion 
between an original message and a reproduction message D for a fixed code rate R. We 
follow the calculation of references QU [161 El EH CEH] . 
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4-1. Replica symmetric solution 

We first assume the replica symmetry (RS). Using the replica symmetric partition 
function Z R s(/3), we find the replica symmetric free energy as 



11 r (3 - f 1 f 1 

—extr Incosh K I dxir(x) / dxTt(x) ln(l + 

2 p 7T,7r L 2 J_! J_x 

+ J2 Pk ^{H \ ^*T(x fc ))Ej[ln(l + (tanh^) f[ 

IS l 1 «/ — 1 7 1 



K ~k=l"~ x ~ fc=l 

— (27 1 C 

+§E p ^)(n/ ^))in(£n[i+^)], (12) 

C c=l^ _1 cr=±lc=l 

where the parameters are determined by the saddle-point equations obtained by 
calculating functional variations: 

C-1 „i c-i 



7r(a;) = ^S^(C)(n / d£ c 7r(£ c ))5(a;-tanh[5]tanh- 1 x c ]), (13) 

C c=l c=l 

*W = zZ^ Pk ^{H / dx k n(x k ))Ej[6(x- (tanh^-J J] a 



(14) 

with K = KPk{K) and C = J2c CPc{C) (See the outline of the derivation in 
Appendix Afl . We can obtain the distortion, which is reproduction errors, u R s{(3) = 



d[Pf RS {P)]/d(3 and the replica symmetric (RS) entropy s RS {(3) = (3[u RS {(3) - f RS {(3)}- 

For arbitrary Pjc(K), Pc{C) and (3, ir(x) = S(x) and n(x) = 5(x) are always 
solutions of the saddle-point equations (IT3"|) and (fT4"j) . These are correspond the 
paramagnetic solution. The paramagnetic free-energy, internal energy and entropy 
are given by fpA R A (P) = \ ~ \ In cosh § - |ln2, u PARA {f3) = | - |tanhf and 
spa R a{/3) = In cosh I — |tanh| + i?ln2, respectively. However, this RS solution 
takes negative entropy while i?ln2 < |tanh| — In cosh |. Especially, when the inverse 
temperature (3 —* 00, the RS entropy becomes s R s{(3) = (R — l)ln2. This means we 
have to look for the true solution beyond the RS ansatz for R < 1. 



4-2. One-step replica symmetry breaking solution 

The replica symmetric breaking (RSB) theory for sparse systems is still under 
development [2U1 ED E21 [25] • Therefore, as a first approach we introduce the 
frozen RSB to produce a solution with non-negative entropy [SI EE21 [13]. The frozen 
RSB method is a limited version of full one-step RSB (1RSB) and includes the RS 
method as a special case. In this 1RSB scheme, n replicas are divided into n/m groups 
which contain m replicas each. The symmetry breaking parameter m was found to be 
m = /3 g //3, where j3 g is a inverse temperature at which the replica symmetric entropy 
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Figure 1. Example of numerical solutions for finite connectivity systems with 
Pk(K) = Sk,2 and Pc(C) = Vc{C)- (a) Rate distortion performance for r = 
0.3, 0.4, • • • , 0.9 (squares). The solid line denotes the rate distortion performance in 
large K and C limit, which coincides with the Shannon bound, (b) Inverse temperature 
j3 g for for r = 0.3, 0.4, • • • , 0.9 (squares). The solid line denotes the inverse temperature 
j3 c , which is defined by spara(Pc) = 0. 



vanishes, i.e., s R s(Pg) = (See Appendix B ). This 1RSB scheme gives the exact solution 
for the random energy model (REM) [6], [26] . For (3 > /3 g , the 1RSB free energy becomes 
fiRSB^P) = fns(Pg)- R can be ragarded as a constant with respect to the inverse 
temperature [3. We assume that the 1RSB scheme is enough good to approximate the 
solution even if K and C are finite. Under this assumption, the distortion D is simply 
given by D = lim^^ u 1RS b{P) = urs(P 9 )- 



5. Results and discussion 



5.1. Basic results 

In large K and C limit, there are no other solutions except ir(x) = 5(x) and 7r(x) = 5(x) 
for the saddle-point equations. We then found the relationship 

R=l-h 2 (D), (15) 

from s RS (f3 g ) = In cosh ^ - § tanh § + R\n2 = and D = u RS (f3 g ) = \~\ tanh 

In finite K and C case, the solutions tc(x) = 5(x) and n(x) = 5(x) also exist, but 
these are no longer stable [6]. We have to solve the equations ([131) and CHI) numerically. 
We choose the proper value of the inverse temperature (3 g which gives s R s(Pg) = by 
using the numerical results of the saddle-point equations. Since the distortion D can 
be evaluated from D = u R s(Pg), we can a ls° obtain the relation between the code rate 
R = K/C and the distortion D in the finite connectivity systems. 

As one of the simplest examples to treat the arbitrary code rate, we here introduce 
degree distributions P K {K) = 5 K , 2 and P C {C) = 7 -^S c , 2 + 2 -^ 1 6 CJ (= V C (C)), which 
are valid for = < r < 1. Here, 5 m ^ n denotes Kronecker's delta taking 1 if m = n and 
otherwise. In this case, we can adjust the code rate R{= r) via the parameter r. 
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Figure 2. Snapshots of the order functions tt(x) and tt(£). (b) a regular case, (a) a 
check-regular and bit-irregular case, and (c) a check-irregular and bit-regular case. 



We apply the Monte-Carlo integration to solve the saddle-point equations. Figure [T] (a) 
shows the rate-distortion performance of this system. Figure [1] (b) shows the inverse 
temperature j3 g , which is a root of the replica symmetric entropy SRs(Pg) = 0. 

5.2. Some typical irregular constructions 

We next apply some degree distributions as typical examples. It should be noted that 
these distributions discussed here are not optimized but heuristically chosen. All three 
examples have the code rate R = 1/2. 

Firstly, we consider a regular case characterized as Pk{K) = 5k,2 and Pc{C) = d~c\4- 
Figure [2] (a) shows stable solutions n(x) and tt(x) of the saddle-point equations for this 
case. It can be confirmed that the functions ir(x) and if(x) are broad in shape. In this 
case, the distortion becomes D = 0.116. The Shannon bound is Dsb = 0.1100. 

Secondly, We treat a check-regular and bit-irregular case whose degree distributions 
are defined as Pk(K) = 5k,2 and Pc(C) = Pc(C), where 

V C (C) = 0.045 c ,i + 0.155 c ,2 + 0.225 Cj3 + 0.225 CA 

+ 0.185 c ,5 + 0.11<y G ,8 + 0.085 c ,7- (16) 

This Vc{C) is a rough approximation of the Poissonian distribution e~ x \ c ~ l / \C — 1)! 
with A = 3. The distortion is D = 0.115 for this case. It represents an ensemble which 
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have at least one non-zero element in each row. In the check-regular case, when we 
choose the non-zero elements randomly, there exists some columns whose elements are 
all zero. In such a situation, the code rate essentially becomes small. 

Lastly, for a check- irregular and bit-regular case, we apply Pk(K) = Vk{K) and 
Pc{C) = 5 C , a, where 

V K (K) = 0.365 K>1 + 0.365 K>2 + 0.20fe, 3 + 0.085 KA . (17) 

This Vk{K) is a rough approximation of the Poissonian distribution e~ x \ K ~ 1 / (K — 1)! 
with A = 1. The reason why we consider this distribution is same to Vc{C). In this 
case, the distortion becomes D = 0.115. These three kinds of distributions give almost 
same distortion. 

Figure [2] (b) and (c) show stable solutions for these irregular cases. It can be 
confirmed that the distribution tc(x) and if(x) become a little bit narrow than the 
regular case. It is considered that the distortion can become small due to this. 

6. Conclusions 

We evaluate typical performance of LDGM codes with irregular bit and check degree 
distributions by applying the replica method under 1RSB ansatz. Our result shows 
that we can use an arbitrary code rate. It might be possible to investigate suboptimal 
irregular degree distributions by using the hill-climbing approach similar to the case of 
the density evolution [27J EE] • 

In the practical point of view, it must be important to evaluate some polynomial 
time encoding algorithms with arbitrary degree distributions. It should be noted that 
the analysis addressed here is based on an exact calculation of the encoder's definition. 
Therefore it can be considered that the distortion obtained by this analysis provides the 
theoretical limit for given check and bit degree distributions. 

Recently, the cavity method was introduced to evaluate the typical performance 
[?]• Since the cavity method does not need the replica trick, it might be able to avoid 
some assumptions. Applying the cavity method to this problem is also important and 
is a part of our future work. 
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Appendix A. Derivation of replica symmetric free energy 

We assume that the event X>^ ... i — 1 occurs independently for every row \i. We then 
have 



= !) = 
p Pf™ = 0) = 1- P/1 , 



(A.l) 
(A.2) 



where P(- ■ •) denotes the probability of the event (■ ■ ■) and p^ = ) ~ K^l/N >*. 
Introducing the constraint concerning the column (jSJ) by using Dirac's delta function, 
the ensemble average over the codes is represented as 

IUK- • ■)] 



eiiw)(eii^ 

[Kp}l*=l {Ci}i=l 
1 AT M 

^MUKe e »u 

4=1 M— 1 <*1=*>«2,--->«K M > 



{A" M } M =l 



{Ct} i=l 



1 ^ f 

2 = 1 



A? 



^■^n n 



2m Z c 



(••■), (A.3) 



where denotes the average over the connectivity parameter. Observing that 
E^E ... ijr \(- - •) = ^-f(X/i(' ' f° r large N, the normalization constant A/© is given 

by ' ' 

AT M 

*=1 A 4 — 1 (11=1,12,— !*iTu) 



(AC)! 



(A.4) 



i=l 



To evaluate the free energy, we calculate the replicated partition function: 

®A,j[Z(P) n ] 



nM/3 

e 2 Eaj 



„ M n 

E «p[|E SW'E4-1) 

L .---.s n M=l (ii.---.iV.. 1 ) a=l 



nM/3 

e 2 



N 

En p *M(EII^(a: 



s ± ,--,s" 



N 



{d} i=l 



^ i=l >/ 1 M=l fri.-.i*-..} 



=1 (ai,-,a m ) 



M=l (ii,-,*K„> 

+ PmE E (cosh^E.^anh^ 
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x E (^c---o---(^c M ---c Al ))- ( A -5) 

We next introduce order parameters g air . ara and g 0) defined by 

1 N 



N 

i=i 



9b = ^E z *- ( A - 7 ) 



i=l 

Using the Fourier expression of the Dirac delta function, we find 

_nM£/ /" dg rf?o\/TT /" dq ai dq ai \ ( t-t /" dq aw ..^ n dq au ...^ r 
= e 2 " ' i i ■ i 



(/^)(n/^)-(n / 

(«i) (ai,-,«n) 
M JV n AT , 

Enw)(Enw)^(n/i^ 

{A^}m=1 {CJi=l ^ * 

X expj-iv|g g + ---+ ^ 9ai,- ) a„5ai 1 -,a„} 

AT AT 

+ g E^ + ---+ E ^,-^E^ 1 •••«?"] 



iM n 

n( T o% M +E e t "(?«.,»* )^)' (A - 8) 



i=l (ai,—,a„) i=l 

M n 

X 

A*=l m=l (ai,-,a m ) 

with T m = (cosh |)™Ej[(tanh 4p) m ]. To proceed further, we introduce the replica- 
symmetric (RS) assumption: 

Qai,-,a m = <I J dx7l(x)x m , (A.9) 

q ai ,-,a m = q J ^dx7r{x)x m , (A.10) 

where n(x) > 0, tx{x) > and J^ 1 dxn(x) = J^ 1 dx7r(x) = 1. This assumption means 
that the order parameters depend only on the number of indices. We write the replica 
symmetric partition function as Z RS ({3)- Using the integral form of the Dirac's delta 
function, we obtain 

E a ,j[Zrs(PT} 

nMS M N 

= s* ^(zn w) (e n povti) 

U {K»}i*=l {Ci}i=l 

x(n{^(n/ ««fe#(*o)(E iid +<*))"}) 

1=1 l " c =l >/ " 1 CT =±1 C=l 
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exp|^— Nqq J dxn(x) J dxfc(x)(l + xx) 
n(^(cosh|r(n / cfafcTr^jEjf^ + Ctanh^IJ 

u=l k=l^- 1 fe=l 



J'fc 



(A.n; 



Finally, substituting this into (TTTT) and taking the limit n — > 0, we arrive at ( 1121) . The 
saddle-point equations (fT3j) and (TH|) are simply obtained as the extremization condition 
of (ED. 



Appendix B. One-step replica symmetry breaking solution 



We follow the calculation of the reference |13j . We assume that the space of configuration 
is divided in n/m groups with m identical configurations in each. 



N 



-s a ■ 



1, if a and (3 are in the same group 
q, otherwise 



(B.l) 



Using this ergodicity breaking assumption, the 1RSB replicated partition function 
becomes 



E 



A,J 



-/3mH(s,J) 



n/mi 



JO 

n/m- 



(B.2) 



n=0 



\n/m" 



= E AtJ [Z RS (pm 
Then we obtain the 1RSB free energy as 

flRSB(f3)= -^E A>J [\jiZ 1RSB (P)] 

pm\o{n/m) 

pm 
= fas ((3m) 

The symmetry breaking parameter m should be determined to extremize the 1RSB free 
energy as 

^hnsB(P) = 0. (B.4) 
The left hand side of this condition becomes 



n/m=0 



(B.3) 



d 

t; — flRSB(P) 
am 



_d_J_ 

dm (3m 



E A j[]nZ RS {(3m)} 
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— E A)J [lnZ RS (i3m)]) 
pm J 



-— s RS {(5m). 
pm 1 



(B.5) 



Namely, the condition (IB.4I) is equivalent to Snsifim) = 0. Therefore, the symmetry 
breaking parameter is given by m = /3 g //3 with s^siPg) — 0. 

References 

[1] T. M. Cover and J. A. Thomas, Elements of Information Theory, 2nd ed., John Wiley and Sons 
Inc., 2006. 

[2] I. Csiszar and J. Korner, Information Theory: Coding Theorems for Discrete Memoryless Systems, 

Academic Press, 1981. 

[3] Y. Matsunaga and H. Yamamoto, IEEE Trans. Inform. Theory, vol. 49, 2225, 2003. 

[4] T. Murayama, Phys. Rev. E, vol. 69, 035105(R), 2004. 

[5] M. J. Wainwright and E. Maneva, Proc. Intl. Sympo. Info. Theory, 1493, 2005. 

[6] T. Murayama and M. Okada, J. Phys. A: Math. Gen., vol. 36, 11123, 2003. 

[7] S. Ciliberti, M. Mezard, J Stat. Meek, vol. 3, 58, 2006. 

[8] S. Ciliberti, M. Mezard and R. Zecchina, Complex systems methods, vol. 3, 58, 2006. 

[9] E. Martinian and M. J. Wainwright, Workshop on Info. Theory and its Applications, 2006. 

[10] A. G. Dimakis, M. J. Wainwright and K. Ramchandran, Info. Theory Workshop, 650, 2005. 

[11] M. J. Wainwright, IEEE Signal Processing Magazine, 47, Sept. 2007. 

[12] Y. Kabashima and D. Saad, Europhys. Lett., vol. 45, 97, 1999. 

[13] R. Vicente, D. Saad and Y. Kabashima, J. Phys. A: Math. Gen., vol. 33, 6527, 2000. 

[14] K. Nakamura, Doctoral thesis, Tokyo Institute of Technology, 2003. 

[15] K. Mimura and M. Okada, Phys. Rev. E, vol. 74, 026108, 2006. 

[16] W. Krauth and M. Mezard, J. Phys. France, vol. 50, 3057, 1989. 

[17] T. Tanaka and D. Saad, Tech. Report, unpublished 2003. 

[18] T. Yano, T. Tanaka and D. Saad, Proc. of SITA2007, 41.4, 2007. 

[19] T. Yano, Doctoral thesis, Keio Univ., (preprint), 2008. 

[20] K. Y. Wong and D. Sherrington, J. Phys. A: Math. Gen., vol. 21, L459, 1988. 

[21] C. de Dominicis and Y. Y. Goldschmidt, J. Phys. A: Math. Gen., vol. 22, L775, 1989. 

[22] P. Lai and Y. Y. Goldschmidt, J. Phys. A: Math. Gen., vol. 23, 3329, 1990. 

[23] Y. Y. Goldschmidt and C. de Deminicis, Phys. Rev. B, vol. 41, 2184, 1990. 

[24] R. Monasson, J. Phys. A: Math. Gen., vol. 31, 513, 1998. 

[25] G. Parisi and F. Tria, Euro. Phys. J. B, vol. 30, 4, 1434, 2002. 

[26] B. Derrida, Phys. Rev. B, vol. 24, 2613, 1981. 

[27] T. J. Richardson and R. L. Urbanke, IEEE Trans. Info. Theory, vol. 47, 2, 599, 2001. 

[28] T. J. Richardson, M. A. Shokrollahi and R. L. Urbanke, IEEE Trans. Info. Theory, vol. 47, 2, 619, 



2001. 



